Interpretation discrepancies of abdominal imaging by on-call radiology residents: Evaluation of risk factors

The aim of this study was to determine the rate, important findings, and risk factors related to discrepancies between on-call residents’ and attending radiologists’ interpretations of abdominal examinations. We identified 1132 eligible patients with abdominal radiology findings that were preliminary interpreted by on-call residents between February 2016 and September 2019. The preliminary interpretations were compared with the final interpretations by abdominal attending radiologists, including clinical data. The preliminary interpretations were analyzed by three radiologists in consensus, who categorized the reports according to organs, important findings (i.e., active bleeding, bowel obstruction, organ ischemia or infarction, and organ rupture), clinical outcomes, and discrepancies with respect to final interpretations. Multiple logistic regression analysis was used to evaluate the risk factors for important discrepant findings. Of 1132 patients, the bowel (n = 567, 50.1%) was the most common organ interpreted by on-call residents, followed by gallbladder/bile duct/pancreas (n = 139, 12.3%) and liver (n = 116, 10.2%). Of 1132, 359 patients (31.7%) had disease with 379 important findings: active bleeding (n = 222), organ rupture (n = 77), bowel obstruction (n = 52), bowel ischemia (n = 24), and organ infarction (n = 4). Sixty-four patients (5.6%) showed discrepancies, and 30 (2.6%) showed 32 important discrepant findings comprising 14 active bleeding, 10 bowel obstructions, 6 organ ruptures, and 2 cases of bowel ischemia. Of the 64 discrepant patients, 33 underwent delayed surgery (n = 18, 28.1%) or interventional treatment (n = 15, 23.4%). In multivariable analysis, bowel obstruction (adjusted odds ratio, 2.52; p = 0.049) was an independent risk factor for determining discrepancy between preliminary and final interpretations. The rate of overall and important discrepancies between on-call residents’ and final interpretations was low. However, given that the bowel was the most frequently interpreted organ, bowel obstruction was identified as a risk factor for discrepant interpretations. The identified risk factor and findings may be useful for residents to minimize discrepancies.


Introduction
In many academic radiology departments, radiology residents often provide after-hour coverage for preliminary independent radiology examinations performed on inpatients and conducted in the emergency department. A common evaluation by on-call radiology residents is conducted on abdominal examinations, which are often challenging, and attending radiologists review these interpretations the next morning.
Many previous studies have reported low rates of discrepancies between the preliminary report from residents and the final report by attending radiologists [1][2][3][4][5][6][7]. Nevertheless, diagnostic errors in preliminary radiology reports may cause discrepancies. Errors are divided into 1) perceptual (misses) errors and 2) interpretation (differential diagnosis) errors [8]. Identifying the underlying risk factors or causes of erroneous evaluation may lead to reduce the discrepancy rate. However, to the best of our knowledge, no study has investigated the risk factors for discrepancies between preliminary abdominal radiology reports provided by residents and the final reports verified by attending radiologists. As misinterpretations during overnight duty may result in changes to treatments and additional evaluations [3], it is important to analyze discrepancies between preliminary and final reports in abdomen radiology studies.
We conducted a retrospective review of preliminary reports of abdominal imaging examinations by radiology residents during after-hour coverage. We analyzed the discrepant cases and risk factors for discrepancies between residents' preliminary reports and attending radiologists' final reports along with the clinical outcomes. If residents can identify such discrepant cases and risk factors for discrepancies and prepare for similar situations before on-call duty, misinterpretations may be reduced, thus improving diagnostic accuracy of preliminary readings.
We aimed to determine the rate, types, important findings, and risk factors related to discrepancies between residents' preliminary reports and final interpretations by abdominal attending radiologists including the clinical outcomes.

Materials and methods
This retrospective study at a tertiary referral center was approved by the Institutional Review Board of Gil medical center (GAIRB2021-378), and the requirement for obtaining written informed patient consent was waived.

Study population
We evaluated 2374 consecutive patients for the preliminary radiology interpretations by oncall radiology residents between February 2016 and September 2019. From this overall set, we identified eligible patients that were over 16 years of age and with a consultation which included abdominal imaging. Among 1180 eligible patients, 16 patients had data recording errors and 32 patients had insufficient follow-up time (less than one month) and were excluded. For this study, we reviewed data on 1132 patients with preliminary radiology interpretations made by on-call radiology residents (Fig 1).

Preliminary report data by resident
In our institution, radiology residents take on-call duty from 5 pm to 8 am on weekdays and 8 am to 8 am overnight on weekends and holidays for emergency department and inpatient examinations. The evaluations are based on the referring clinician's questions made by phone call regarding simple radiography, ultrasonography (US), CT, and MRI. The questions regarding simple radiography, CT, and MRI are interpretative, and those on US query about the possibility of the resident to perform on-call US and interpret it. The residents in our institution begin taking on-call duty responsibilities between the second half of the first year and the first half of the fourth year of training. The duty consists of mainly second-and third-year job (approximately 85-90%) and remnant job of the second half of the first year and the first half of the fourth year. All residents are educated about abdominal radiology for more than 8 weeks before taking a call. In some cases, when a junior (first-year) resident has difficulty with a case during on-call duty, they can ask a senior resident regarding the case. At the end of each

PLOS ONE
Discrepancies abdominal on-call radiology overnight shift, all residents should record a list containing the consulted patients, their information, and preliminary reports for educational purposes in our institution's database. The database includes the patients' sex, age, number, image study date, date of duty, name of oncall duty resident, and reason for the consultation.

Evaluating discrepancies
The complete database was reviewed retrospectively by three radiologists with experience in abdominal radiology in consensus reading. One radiologist had 10 years of experience (S.H.P) and the remaining two radiologists (S.J.Y., H.J.L.) had 3 years of experience at the time of the study. The data were classified according to specific organs, examination types, resident's grade (i.e., years of training), presence of important findings, and discrepancies with and without legal consequences. Important findings of the abdomen were defined as the presence of a potentially life-threatening condition that may require immediate clinical management [9,10]: 1) presence of active bleeding, 2) bowel obstruction, 3) organ ischemia or infarction, or 4) organ rupture based on the modification of critical results in abdominal radiology [11][12][13][14][15][16][17]. The definitions and descriptions of findings 1-4 are summarized in S1 Table. The preliminary reports were evaluated for discrepancies of the final interpretations, including final reports and clinical outcome (surgery with pathology, intervention, endoscopy, and medical treatment based on EMR). The final reports were completed within 1-2 days after the preliminary report by one of four abdominal attending radiologists. The clinical outcome was reviewed on electronic medical records and classified as surgery, interventional treatment, endoscopic procedure, or medical treatment with clinical follow-up. The final reports with clinical outcome (i.e., final interpretations) were used as the reference standard for resident on-call reading.
MRI examinations were performed for evaluation of acute appendicitis in pregnant patients and magnetic resonance cholangiopancreatography using a 3T scanner (Skyra, Siemens Healthineers).
Appendix US and upper abdomen US were included in the US examination.

Statistical analysis
Residents were grouped as "discrepant" when the preliminary report differed from the final interpretation and as "identical" when the reports agreed. Patient characteristics in each group were compared using Student's t-test and chi-square test. Univariable and multivariable logistic regression analyses were used to evaluate the risk factors of discrepancy interpretations by on-call residents, adjusting for covariates. Parameters with a p value less than 0.2 on univariable analysis, were included in the multivariable analysis [18,19]. Multivariate logistic regression analysis was performed using the backward likelihood ratio. Differences were considered statistically significant with a 95% confidence interval and p < 0.050. All statistical analyses were performed using the SPSS software (version 22.0, IBM).

Characteristics of preliminary reports with important findings
Only thirty patients (2.6%) showed 32 important discrepant findings, including 14 active bleeding, ten bowel obstructions, six organ ruptures, and two bowel ischemia. Two patients had two important findings. Of 30, all cases were CT, initial CT examinations were 16 (53.3%), and bowel was the most common organ (20, 66.7%). Although 14 patients showed active bleeding on CT scans, the on-call residents were unable to detect it. In addition, 10 cases of bowel obstruction were mistaken as paralytic ileus (6 cases), pelvic inflammatory disease (1 case), absence of bowel perforation (1 case), acute diverticulitis (1 case), and paraduodenal hernia (1 case) in the preliminary reports. In the patient with paraduodenal hernia, the resident detected the transitional zone of the small bowel, but different diagnosis was interpreted during on-call duty. Therefore, of the 30 discrepant cases, 23 (76.7%) were perceptual errors (i.e., no detection of the transitional zone at bowel obstruction or active bleeding focus in preliminary readings) leading to misinterpretations. Perceptual errors were most frequently noted in preliminary reports with important discrepant findings. Risk factors for predicting discrepancy interpretations by on-call residents. Table 5 shows risk factors for discrepancy between preliminary and final interpretations in the important findings (n = 359). The results of the univariable analysis showed a specific grade of residents (resident 2 nd year), and bowel obstruction (p < 0.2) were available risk factors included in the multivariable analysis. In the multivariable analysis, bowel obstruction (adjusted OR, 2.52; 95% CI: 1.00-6.50, p = 0.049) was an independent risk factor for important discrepant findings (Fig 3). Of 52 bowel obstruction interpretations, eight were interpreted by 1 st year (three discrepancies), 18 were interpreted by 2 nd year (four discrepancies), 21 were interpreted

Discussion
This study investigated the rate and risk factors of discrepancies between on-call residents' and final interpretations considering the attending radiologist's report and clinical outcomes on abdominal examinations. The rate of overall and important discrepancies was low in abdominal radiology. Bowel obstruction was a significant risk factor for important discrepant findings. The bowel showed the highest discrepancy. Educating abdominal residents emphasizing the bowel and bowel obstruction may improve the interpretation ability of radiology reports during on-call duty. We found a 5.6% (64/1132) discrepancy rate between preliminary and final interpretations. Previous studies have reported an overall discrepancy rate from 0.1% to 3.8% [1][2][3][4][5][6][7][20][21][22][23][24][25] in the preliminary radiology reports. Few studies [1,3,20] have reported that body CT may be associated with discrepant interpretations given the slightly higher discrepancy rate (6.4%, 9.8%, respectively) compared with the overall discrepancy rate [1,3]. Our study analyzed abdominal cases, mainly abdominopelvic CT. Our rate (5.6%) was similar or slightly lower than that reported in previous studies. We consider that the resulting rates may reflect different clinical practice environments or educational efforts regarding the review of discrepant cases.
The bowel was the most common preliminary interpreted organ and had the highest discrepancy rate in our study. The high frequency of bowel interpretations during on-call duty may be explained by the common pathologies of acute abdominal pain, including gastrointestinal perforation or inflammation and bowel obstruction or infarction [26]. This finding was similar to that in a previous study regarding abdominal and pelvic CT taken in an emergency department, where bowel disease showed the highest discrepancy between preliminary and final reports [25]. Another study suggested that acute appendicitis in contrast-enhanced abdominopelvic CT was the most common cause of misinterpretation [20]. Considering previous studies and our present study, residents should pay urgent attention to the evaluation of bowel disease during on-call duty and study radiologic findings of this pathology before starting and during their after-hour coverage.   We found that bowel obstruction was significantly associated with discrepant preliminary and final interpretations, with an adjusted OR of 2.52. Abdominal CT is an important diagnostic modality for detecting small bowel obstruction and predicting surgical candidates [27,28]. The CT findings of small bowel obstruction were feces signs, transitional zones, beak signs, mesenteric vessel course, presence of closed-loop obstruction or ischemia, and ascites [12,28,29]. The radiologist can detect the transitional zone between the dilated and collapsed loops using a bowel trace on consecutive CT images. One possible explanation for our results is that the bowel tracing skills to find the transitional zone (i.e., obstruction site) are acquired through a relatively long learning curve, which may have affected the preliminary report results. Our results also showed misinterpretations by 1 st and 2 nd year residents were higher than those by 3 rd and 4 th year residents. Although next-day CT readings by the abdominal attending radiologist can minimize the patient severity risk, performing early accurate diagnosis of bowel obstruction on the preliminary report may improve the patient care because delayed surgical management of bowel obstruction can increase the mortality and morbidity rates of patients and prolong hospitalization [30]. Additional practice before and during on-call duty is thus essential to identify the number and location as well as the presence of transitional zones related to closed-loop small bowel obstruction and development of pneumoperitoneum, pneumatosis intestinalis, and portal vein gas, which is highly suspected to be a surgical candidate and complications of bowel obstruction [28].
Among the 1132 evaluated cases, our results showed 30 important discrepant findings categorized into active bleeding, bowel obstruction, organ ischemia or infarction, and organ rupture. We found that perceptual errors during preliminary interpretation were the most common cause of important discrepancies. Perceptual errors develop during initial screening (i.e., failure to recognize an abnormality) and cause missed diagnoses in radiology. Consistent with our results, perception errors have been reported to be the most common and important mistake made by radiologists [8,31,32]. We suggest residents to collect and review missed lesions showing important findings on CT to reduce the error incidence and improve the diagnostic accuracy. We believe that education can improve radiologic interpretations throughout training. Critical point of the important findings obtained with imaging modalities may require surgical or interventional approaches. As important discrepant findings are directly related to life-threatening scenarios, our educational goal should be aimed at reducing the frequency of discrepancies. Our study further showed that 446 patients (39.4%) underwent surgery or intervention. Among them, 33 patients (2.9%) underwent delayed surgery or interventional treatment after a preliminary radiology report. These results suggest that the discrepancy in on-call residents' preliminary interpretations can lead to management changes. Similarly, previous studies have demonstrated that discrepancies in on-call residents' preliminary interpretations can affect patient care and management [3,[20][21][22][23][24]. McWilliams et al. [22] studied abdominal imaging and other body-part imaging, finding that 44.6% of the discrepant preliminary cases resulted in management changes, and 14% of the discrepant preliminary cases caused therapeutic management changes, such as surgery and interventional endoscopic procedures, while 11.9% of the discharged patients were recalled. Ruchman et al. [20] suggested that 7.2% of discrepant reports showed a negative effect on patients. Friedman et al. [24] reported that 35.7% of such cases increased the patients' morbidity and hospitalization period, whereas discrepant preliminary reports did not increase mortality or long-term outcomes.
We also found that the experience of a second-year resident was a possible risk factor for the discrepancies. However, a specific training degree was not a significant risk factor after multivariable analysis. In our study, it was difficult to evaluate the experience of a first-year resident because the number of overnight duty days in the first-year was small and they can ask a senior resident regarding the difficult case. These results may differ from a previous study [20], which showed the highest discrepancy rate for residents who were in their third year of training. Mellnick et al. [5] reported that a higher grade of residents led to more discrepancies, whereas other studies reported that a higher grade of residents led to reduced discrepancies [1,3,6,7]. We suggest that the overnight coverage ratio in a specific residency year and different education systems depending on the academic institutions can affect the discrepancy results. In addition, training programs have undergone many changes over the years, including strict work-hour regulations in South Korea, increased training under supervision, and decreased trainee independence [7,33]. Few studies have reported higher error rates in residents working more than 10 consecutive hours overnight [34] and increasing their caseload or working hours [35], and these error rates may be associated with fatigue or circadian effects. This study has various limitations. First, it was a retrospective single-center study in South Korean population that inevitably leads to selection bias. Second, our study was performed in a tertiary academic medical institution including regional emergency medical, cancer, and trauma centers for a specific region, possibly impacting the severity of cases in enrolled patients. Third, discrepancies were noted only for a small portion of patients. Thus, our study revealed one risk factor. Including more patients may be conducive to identify additional risk factors. Finally, only a small number of imaging modalities besides CT were considered in this study.
In conclusion, overall and important discrepant findings between preliminary interpretations by on-call residents and final interpretations showed a low rate in abdominal radiology. Nevertheless, bowel obstruction is a risk factor for discrepancies, and the bowel is the most common target of on-call interpretations.